> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/mlfoundations/open_clip/llms.txt
> Use this file to discover all available pages before exploring further.

# get_tokenizer

> Get the appropriate tokenizer for a CLIP model

## Overview

Returns a tokenizer instance based on the model name or identifier schema. Automatically selects the appropriate tokenizer type (HFTokenizer, SigLipTokenizer, or SimpleTokenizer) based on the model configuration.

## Function Signature

```python theme={null}
def get_tokenizer(
    model_name: str = '',
    context_length: Optional[int] = None,
    cache_dir: Optional[str] = None,
    **kwargs
) -> Union[SimpleTokenizer, HFTokenizer, SigLipTokenizer]
```

## Parameters

<ParamField path="model_name" type="str" default="''">
  Model identifier that determines which tokenizer to use. Supports multiple schemas:

  * `'ViT-B-32'`: Built-in model name (looks up config)
  * `'hf-hub:org/repo'`: Load from Hugging Face Hub
  * `'local-dir:/path/to/folder'`: Load from local directory
</ParamField>

<ParamField path="context_length" type="int" default="None">
  Maximum sequence length for tokenization. If None, uses the value from model config or defaults to 77.
</ParamField>

<ParamField path="cache_dir" type="str" default="None">
  Directory to cache downloaded tokenizer files when loading from Hugging Face Hub.
</ParamField>

<ParamField path="**kwargs" type="dict">
  Additional tokenizer-specific keyword arguments passed to the tokenizer constructor. Overrides config values.
</ParamField>

## Returns

<ResponseField name="tokenizer" type="Union[SimpleTokenizer, HFTokenizer, SigLipTokenizer]">
  Tokenizer instance appropriate for the specified model:

  * **SimpleTokenizer**: Default OpenAI CLIP tokenizer (BPE-based)
  * **HFTokenizer**: Hugging Face transformers tokenizer wrapper
  * **SigLipTokenizer**: SigLIP T5-compatible sentencepiece tokenizer
</ResponseField>

## Examples

### Get tokenizer for built-in model

```python theme={null}
import open_clip

# Get default tokenizer for ViT-B-32
tokenizer = open_clip.get_tokenizer('ViT-B-32')

# Tokenize text
texts = ["a photo of a cat", "a photo of a dog"]
tokens = tokenizer(texts)
print(tokens.shape)  # torch.Size([2, 77])
```

### Load from Hugging Face Hub

```python theme={null}
# Load tokenizer from HF Hub model
tokenizer = open_clip.get_tokenizer(
    'hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K',
    cache_dir='./hf_cache'
)
```

### Load from local directory

```python theme={null}
# Load from local model directory
tokenizer = open_clip.get_tokenizer(
    'local-dir:/path/to/model',
    context_length=77
)
```

### Custom context length

```python theme={null}
# Override default context length
tokenizer = open_clip.get_tokenizer(
    'ViT-L-14',
    context_length=128  # Longer sequences
)
```

### Pass custom tokenizer kwargs

```python theme={null}
# Pass additional tokenizer arguments
tokenizer = open_clip.get_tokenizer(
    'hf-hub:openai/clip-vit-base-patch32',
    clean='canonicalize',  # Custom text cleaning
    additional_special_tokens=['<mask>']  # Add special tokens
)
```

## Tokenizer Selection Logic

The function automatically selects the tokenizer type based on:

1. **HFTokenizer**: Used if `hf_tokenizer_name` is specified in the model's text config
2. **SigLipTokenizer**: Used for models with 'siglip' in the name (when no HF tokenizer specified)
3. **SimpleTokenizer**: Default fallback for OpenAI CLIP models

## Schema Support

<CardGroup cols={3}>
  <Card title="Built-in" icon="database">
    Use model names from OpenCLIP's model registry
  </Card>

  <Card title="HF Hub" icon="cloud">
    Load from Hugging Face Hub: `hf-hub:org/repo`
  </Card>

  <Card title="Local" icon="folder">
    Load from local directory: `local-dir:/path`
  </Card>
</CardGroup>

## Notes

* For `local-dir` schema, an `open_clip_config.json` file must exist in the directory
* Context length priority: function argument > model config > default (77)
* Tokenizer kwargs from the function call override those in the model config
* If model config cannot be loaded, falls back to SimpleTokenizer with default settings